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(54) Process for displaying movable bodies. 

(57) The invention concerns a process for displaying movable bodies wherein an optical 
display of the body and a data field subordinated to the body are displayed simultaneously or 
alternately in real time. The procedure consists of the following stages: 

- Providing an optical display system consisting of camera and monitor; 

- Allocation of a spatial data field to the body located in a particular position; 

- Continued recording of the spatial position of the body; 

- Continued calculation of a display of the data fields, which corresponds to all given 
positions of the body;- 

- Simultaneous or alternating display of the optical image and the data field on the monitor. 
The invention further involves a device to implement the procedure. 
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The invention concerns a procedure for the display of movable bodies. 

In technology and medicine it is frequently necessary to enhance the optical display of objects 
with additional information or to combine different displays of an object with each other. 

Thus, for example, when performing certain operations on the human body, standard practice 
requires comparing pre-operative images with intraoperative images. These images can be 
two-dimensional X-ray images, tomograms, ultrasound displays orNMRs. 

In order to bring two such displays into agreement, it is now possible to identify a certain 
number of anatomically characteristic points and ascertain their coordinates in the two 
displays. This allows calculation of the six parameters of a solid body transformation, i.e. the 
three parameters of translation and three parameters of rotation, and also a scaling factor as 
appropriate. It is thus possible to rotate and display one of the two displays so that two 
pictures can be displayed from the same perspective and on the same scale. This process is 
known as "matching." 

In order to simplify matching, it is suggested to place on the object to be investigated certain 
markings which are visible in both displays and are easily identifiable. These may be probes 
made of a suitable material, whose position on X-ray pictures can be readily ascertained. In 
the case of optical displays, these may be color markings on the surface of the skin. Here too 
only an ex post display of two frames is possible. 

In order to combine an optical display of an object with other displays - which, as mentioned 
above may be two-dimensional X-ray images, tomograms, ultrasound displays or NMRs or 
similar -it is first possible, in principle, to create and store a three-dimensional model of the 
surface of the body. This model can subsequently be matched in the above-mentioned fashion 
with the data of an NMR. This enables optical display of the body from any desired 
perspective and simultaneous insertion of data from tomography. The optical display is 
calculated from the model. 

Such a procedure is extremely costly, since it requires a great deal of storage space, as well as 
large-scale computing capacity. In addition, real-time display is not possible. 

From AT-B 384 544 a procedure is known for determining the position of body parts. 
Ultrasound sensors are used to determine the relevant position in order to ascertain the 
mobility of joints. However, this is a purely mathematical process which does not allow for 
any showing of images and hence cannot be displayed. 

The same drawback applies also to the device described in DE-C 34 06 179. 

US-A 4, 197,855 describes a device to measure the movement of a human lower jaw, where 
an angular permanent magnet is used to determine spatial position. Here too, no further 
imaging processing is envisaged. 

From DE-A 38 07 578 a process is known for measuring individual points on a human head 
by means of video cameras. The points to be measured are pictured in a vertical image and 
recorded electronically together with reference points placed on the head. An optical display is 
not envisaged even for vertical frames. This is even more applicable to real-time display. 

The object of the invention is to avoid these drawbacks and create a procedure which will 
make it possible, in a simple fashion, to combine optical displays with other data graphically 
and display them clearly. 
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In the procedure according to the invention, provision is made for an optical display of the 
body and a data field allocated to the body to be displayed simultaneously or alternately from 
the same perspective and to the same scale in real time. This means that the body can move 
freely in the image area. As it does so. it is displayed in real time, i.e. simultaneously, on a 
display device. A stored data field accompanies the movement of the body and can also be 
displayed. In this way a major degree of clarity is achieved. 

The procedure according to the invention is preferentially carried out in the following steps: 

- Preparing an optical imaging system consisting of a camera and a monitor; 

- Allocating a spatial data field to a body located in a specific position; 

- Continued recording of the spatial position of the body; 

- Continued calculation of a display of the data field, which at any one time corresponds to the 
position of the body; 

- Simultaneous or alternating display of the optical image and the data field on the monitor. 

It is possible to use as data field the position of anatomically characteristic points, an X-ray 
tomograph, an NMR, an ultrasound image or similar. 

The position of anatomically characteristic points can be recorded by means of a device such 
as a 3D digital stylus, i.e. a magnetic stylus. The tip of the stylus is used to mark the relevant 
points; at the same time, activation of the Enter key triggers the determination of the precise 
position. Obviously, only points on the surface of the body can be marked. 

If points which lie inside the body, such as characteristic points of bones, are to be used as 
data points, these should be determined by a method such as stereophotometric measurement 
on X-ray images. An instance of the application of this approach would be the measuring of 
the movements of joints. To do this, two or more X-ray pictures of the corresponding limbs 
are first taken, using sensors placed on the surface of the skin. The focal planes of these 
pictures are preferably at right angles to each other, enabling the coordinates of characteristic 
points to be determined. 

A preferential variant of the procedure involves a 3D sensor - preferably a magnetic sensor - 
which is securely connected to the body and records the position of the body, making it a 
simple matter to continuously determine the position of the body. 

The data field does not, however, necessarily have to be restricted to the position of points of 
the body. An X-ray image can be used as a data field. The position of the image relative to the 
body is determined by a 3D sensor which is securely 'connected to the body during the X-ray 
imaging, with an additional such sensor being connected to the X-ray plate holder. The normal 
two-dimensional X-ray image is assigned to a plane outside the body, e.g. the plane in which 
the X-ray was located when the picture was taken. This X-ray image then represents the data 
field and it is possible to display the body of a patient with this X-ray picture in such a way as 
if there was a firm link between it and the X-ray picture. 

It is similarly possible in to use as data field an ultrasound display, whose position relative to 
the body is determined by having a 3D sensor, preferably a magnetic sensor, which during the 
exposure to ultrasound waves is securely connected to the body. Such a sensor consists of a 
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small magnet which enables not only its spatial position but also its relative position to be 
determined. 

It is possible for the optical image to be superimposed with the display of the data field on the 
screen. At this point, it is possible for this data field to be inserted in the optical display. 
However, it is also possible to toggle between the optical display and the data display. This 
offers the user diverse options so as to have both a graphic display and also an optimum basis 
for exactly measuring body parts. 

It is particularly advantageous if the optical image and the display of the data field on the 
screen are displayed in different windows. The user can move, superimpose, zoom, or blank 
out these windows in the known fashion. 

In a particularly favorable embodiment of the procedure according to the invention, it would 
be possible for part of the displayable data field to consist of an imaginary axis selected 
arbitrarily and for this axis to be obtained by the following steps: 

- Displaying the body as a vertical image in at least two positions; 

- Providing the possibility of the user drawing the image of the axis into the vertical images; 

- Calculating the spatial position of the axis; 

- Joint display of the axis together with the moving bodies displayed in real time. 

If, for example, the axis of the femur is to be inserted into the image of a human thigh, the 
following procedure will be followed: First the position sensor is stuck to a part of the skin 
which. is influenced as little as possible by movements of soft parts. Frames are then taken 
using the video camera and displayed simultaneously or consecutively on the screen. Because 
of the position sensor, when doing this the position of the body part is known at the time that 
the picture was taken. The user can then use the mouse to draw an axis into each individual 
shot. For the computer, this constitutes the projection onto the focal plane of an axis located in 
the three-dimensional space. If this axis has been displayed in two different views, e.g. a front 
view and a side view, the spatial position can then be determined. 

In this context, it has proven to be advantageous for some four individual shots to be 
displayed side by side, each in a window of its own. The user takes each of these images and 
uses them to enter or modify the axis. Once the spatial position of the axis can be determined, 
this is also calculated and displayed for the other images. This makes possible accurate control 
of input. 

It is similarly possible for part of the displayable data field to consist of a system of 
coordinates which can be chosen arbitrarily, their axes being obtained in the fashion described 
above. After the spatial determination of an axis of an orthogonal system of coordinates, only 
one additional axis must be plotted in an image in order to define the position. 

After input, the program calculates the position of the axis or axes relative to the position 
sensor so that it is able to insert these axes into every display of the body part, in order for 
these to move in step with the body part. 

The inserted axes or the system of coordinates can independently display the data field as 
defined in the invention, or can be displayed in addition to an X-ray tomograph or similar. 
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The invention further concerns a device for the display of movable bodies. According to the 
invention, this device consists of: 

- a camera; 

- a monitor; 

- a position sensor which is securely connected to the body to be displayed and enables the 
relevant position on the body to be determined; 

- means which enable either simultaneous or alternating display of the optical image and data 
field on the monitor. 

These above-mentioned means are generally a computer which performs the requisite 
calculations. These calculations must take account of the spatial distance between the focus of 
the camera optics and the starting point of the field used for the position sensor, i.e. the 
magnetic field generated by a magnetic field emitter. The calculation also includes the relative 
position of the points of the data field relative to the position sensor. However, in performing 
these calculations the computer can be replaced by an appropriately programmed 
microprocessor. 

A particularly simply and easy-to-achieve embodiment of the invention can be realized by 
using a monitor as a PC screen, the camera being connected to the video input and the data 
field being input into another input or port. In this setup, the data stream resulting from the 
picture of the optical images completely bypasses the computer and therefore does not tie up 
its processing capacity. As a result, all of its computing power is available for additional 
display of the data field. 

The invention is described below in greater detail by means of an embodiment shown in the 
figure. 

A video camera 1 is securely connected to a magnetic field emitter 2. This magnetic field 
emitter 2 consists primarily of a coil which is connected to an electrical circuit and generates a 
magnetic field. A magnetic sensor 4 is located at the patient's head 3. This magnetic sensor 
also consists of coils in which the field generated by the magnetic field emitter 2 induces 
currents. By measuring these currents, it is possible to precisely determine the position of the 
magnetic sensor 4 to within around one millimeter. 

The magnetic stylus 5 is used to mark three measurement points A, B, and C. As in the 
magnetic sensor 4, coils are fitted on the magnetic stylus 5 for position-recognition purposes. 
The marking is carried out by the tip 6 of the magnetic stylus being placed on the appropriate 
point and a release button then being activated. The computer stores the transient position of 
the tip 6 as a data point. The head 3 is displayed together with the data points A, B, and C on 
the monitor 8 of the computer 7. This makes it possible to display these data points, even if 
they would not be visible as real points because they are on the side of the head which is 
turned away from the camera. The position of other concealed points can also be displayed by 
means of this method. Thus the magnetic stylus 5 can be used to mark the position of a 
specific tooth which can be localized when the patient's mouth is closed. If this tooth is in the 
lower set, the position of the lower jaw must, however, also be recorded. This is made 
possible by applying another sensor (not shown). 
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If an additional magnetic sensor 9 is applied on camera 1. it is not necessary for there to be a 
rigid link between camera 1 and magnetic field emitter 2. The computer 7 will then take 
account of the variable displacement between camera 1 and magnetic field emitter 2. 

Patent claims 

1 . Procedure for displaying movable bodies, in which a visual display of the body and a data 
field assigned to the body can be shown simultaneously or alternately in real time, from 
the same perspective and to the same scale. 


2. Procedure for displaying movable bodies, consisting of the following steps: 

- Preparing an optical imaging system consisting of camera and monitor; 

- Allocating a spatial data field to a body located in a specific position; 

- Continued recording of the spatial position of the body; 

- Continued calculation of a display of the data field, which at any one time corresponds 
to the position of the body; 

- Simultaneous or alternating display of the optical image and the data field on the 
monitor. 


3. Procedure for display of a human or animal body according to either of Claims 1 or 2, 

wherein the data field used consists of the position of anatomically characteristic point, an 
X-ray tomograph, NMR, ultrasound display, or similar. 


4. Procedure according to any of Claims 1 through 3, wherein, in order to record the 

position of the body, use is made of a 3D sensor, preferentially a magnetic sensor, which 
is securely connected to the body. 


5. Procedure according to Claim 4, wherein the data field used consists of an X-ray image, 
whose position to the body is determined by the fact that during the X-ray picture a 3D 
sensor is securely connected to the body and another such sensor is connected to the 
X-ray plate holder. 


6. Procedure according to Claim 4, wherein the data field used consists of an ultrasound 

display, whose position to the body is determined by the fact that during the X-ray picture 
a 3D sensor is securely connected to the body and another such sensor is connected to the 
X-ray plate holder. 
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7. Procedure according to any of Claims 1 through 6 ? wherein the optical image is 
superimposed with the data field on the screen. 


8. Procedure according to any of Claims 1 through 6, wherein the optical image and the 
display of the data field on the screen are shown in different windows. 


9. Procedure according to any of Claims 1 through 6, wherein part of the displayable data 
field consists of an imaginary axis selected arbitrarily, this axis being obtained by the 
following steps: 

- Displaying the body as a vertical image in at least two positions; 

- Providing the possibility of the user drawing the image of the axis into the vertical 
images; 

- Calculating the spatial position of the axis; 

- Joint display of the axis together with the body displayed moving (?) in real time. 


1 0. Procedure according to Claim 9, wherein part of the displayable data field consists of a 
system of coordinates selected arbitrarily, whose axes are obtained in the fashion 
defined in Claim 9. 

1 1 . Device for displaying movable bodies, consisting of: 

- a camera (1); 

- a monitor (8); 

- a position sensor (4) which is securely connected to the body (3) to be displayed and 
enables the relevant position of the body (3) to be determined; 

- means which enable the optical image and data field to be displayed on the monitor 
(8) either simultaneously or alternately. 


12. Device according to Claim 1 1 , wherein the screen of a personal computer (7) is used as 
monitor, the camera (1) being connected to the video input and the data field being input via 
the personal computer (7) into another input or port. 


Quality Translations Job No. 32815 


18 


